Software applications including system software and application software are often developed through stages. For example, a software application may first be coded and built as an executable software application including certain basic features. Subsequently, the code may be modified and augmented with more features and bug fixes. The modified and augmented codes may then be built as an updated version of the software application. This process may continue through the life cycle of the software application, including major revisions and minor updates/fixes. In practice, each update or revision of the software application may be a version of the product that may be referred to as a build of the software application. A build of a software application can be a released or unreleased version of the software application, and can be the executable code compiled from source codes of the software application or can be scripts. Thus, a software application may have undergone multiple builds during its life cycle.
A software application may be tested before being released as an executable version of the software application. The software application may or may not be thoroughly performance tested prior to the release. For example, if the modification to the software application is designed to fix a particular bug, the test before the release of the build may be limited to the specific bug rather than a full performance test. In other occasions, a build may include major updates to the software application. Under this scenario, all aspects of the software application performance may be tested. Software performance testing may include load testing, stress testing, endurance testing, and spike testing. The test results may include the measurement of a set of performance parameters while executing the software application. The set of performance parameters may include, e.g., server response time, computational resource (such as CPU) consumption, and throughput of a system, under different performance testing scenarios. Although software applications are tested before being released, modern software applications are so complicated that any performance test can only be achieved to the extent that testing resources are allowed and by no means exhaustive. Certain aspects of the performance testing may have been missed due to noise in the testing results. Thus, certain performance degradations may not exhibit in the immediate build that includes the code causing the performance degradation. Instead, the performance degradation exhibited may have been caused by the modifications done in a much earlier build.
Regression testing has been used to determine which build out of a series of builds causes the performance degradation. Namely, whenever a performance degradation or bug is uncovered for a new version of a software application, a bisect method is used to determine which build of the software application is the real culprit of the performance degradation or bug. Assuming a certain prior build is known to be reliable (e.g., a build that has been thoroughly tested), the bisect method may include executing performance tests on a build in the middle between the known reliable build and the newest build of the software application to determine if the tested build is problematic. The bisect method is then repeated between the known reliable and the middle build if the performance of the middle build shows degradations, or the bisect method is repeated between the middle build and the newest build if the performance of the middle build shows no degradation.
However, under certain circumstances, the bisect method does not reliably identify the problematic build when the performance test results contain noise. Due to inaccuracy in test performance measurements, insufficient test time, and unpredictable factors during the execution of the software application, the test results may be inconsistent, or even worse, it sometimes may present contrary results. The noise in the performance test results may lead to a failure to identify the problematic build.